Reassembly engines for multilink applications

ABSTRACT

Methods, systems, and apparatus for processing packet fragments for reassembly into packets. Datastreams containing packet fragments are received and data corresponding to different channels are segregated. For each channel, data is then aggregated to recreate packet fragments. Once the fragments are recreated, they are, if required, resequenced using a two-tier memory mapping structure where only pointers, and not the fragments, are manipulated. After resequencing fragments to reassemble the packet, the reassembled packet is dispatched to its ultimate destination. The present invention also provides a method of load balancing between multiple links when packet fragments are transmitted from a fragmenting node through a multiple link bundle.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 09/934,847 filed on Aug. 23, 2001, which is hereby incorporatedby reference.

FIELD OF THE INVENTION

The present invention relates to computer networking and, morespecifically, to applications which use multiple links between nodes asa single link.

BACKGROUND TO THE INVENTION

The explosion of interest and reliance on the Internet has led to anincreasing demand for increased data carrying capacity in networks. Thenigh ubiquitous T1 standard is beginning to show its inadequacies forsome applications. Unfortunately, users who need more than the T1's1.544 mbps capacity used to have no recourse but to upgrade to the T3standard operating at 44.736 mbps. This jump can be inconvenient forusers who are only willing to pay for, at most, 20-30 mbps. One solutionto the problem was the concept of “bundling” multiple T1 lines to arriveat higher speeds and capacities. This solution essentially involvesaggregating multiple T1 lines or links and using them as a single linkfrom the perspective of providing bandwidth.

While the “bundling” solution seemed to solve the problem theoretically,technical issues arose when implementing the solution. To optimize thebandwidth in the multiple link bundle, multiple link protocols (such asFRF.12/16 and RFC 1990/2686) were developed. These protocols called fordividing or fragmenting packets (units of data sent from one networknode to another) into smaller elements. These fragments are thentransferred across the multiple links simultaneously. This maximizes thelink bandwidth available.

As an example, if a single packet takes x units of time to transfer overa single link, it should theoretically be possible to divide the packetinto 5 smaller elements and transmit these five elements over 5identical links. Such a transmissions should only take x/5 units oftime. Of course, the above example ignores the overhead cost offragmenting and, later on, reassembling the packet. It is thisreassembly problem that can provide the most issues in implementing the“bundling” solution. Essentially, the system which implements the linkbundling solution must:

1) transmit packets across the member links in the multiple link bundle;

2) fragment the packets while transmitting across the bundle to reduceskew and delay for packet transfers;

3) support multiple simultaneous fragmentation sessions; (this is neededto support multiple independent multiple link bundles and also multipleindependent multiple link sessions within a single bundle); and

4) receive data from all the links, reassemble the data into fragments,and then into packets.

Current solutions for the reassembly problem have their own drawbacks.U.S. Pat. No. 5,590,122 issued to Sandorfi et al and U.S. Pat. No.5,396,490 issued to White et al have proposed solutions directed atpacket reassembly but they do not specifically address multiple linkapplications. Other solutions suffer from the following drawbacks:

a) a low number of bundles supported due to the processing requirementsof the bundle reassembly;

b) supporting only a single reassembly session per bundle;

c) supporting a small number of aggregate bundles due to processing andmemory limitations;

d) an assumption that all links operate at the same speed to simplifythe processing; and

e) line rate processing of all small packets is often limited to theoverhead associated with processing each packet.

From the above, there is therefore a need for a solution that can, amongother things, support multiple reassembly sessions, support largenumbers of bundles and a large number of channels within those bundles,and also be transparent to differing link speeds.

SUMMARY OF THE INVENTION

The present invention overcomes the deficiencies of the prior art byproviding methods, systems, and apparatus for processing packetfragments received for example from a bundle of links for reassemblyinto packets. Datastreams containing packet fragments are received anddata corresponding to different channels are segregated. For eachchannel, data is then aggregated to recreate packet fragments. Once thefragments are recreated, they are, if required, resequenced using atwo-tier memory mapping structure where only pointers, and not thefragments, are manipulated. After resequencing fragments to reassemblethe packet, the reassembled packet is dispatched to its ultimatedestination. The present invention also provides a method of loadbalancing between multiple links when packet fragments are transmittedfrom a fragmenting node through a multiple link bundle.

In a first aspect, the present invention provides a method of reorderingpacket fragments, said packet fragments being derived from a packetwhich has been divided into at least two fragments, each of said atleast two fragments having a sequence number denoting a position in aproper sequence for correct ordering of each fragment for a reassemblyof said packet, each fragment being represented by a head pointer, themethod comprising:

a) receiving a received head pointer for a received fragment;

b) determining a sequence number for said received fragment;

c) determining a first slot position for said received head pointer in afirst tier pointer array, said first slot position being determined bythe sequence number for said received fragment, said first slot positionhaving a tier pointer pointing to a second tier pointer array;

d) determining a second slot position for said received head pointer inthe second tier pointer array, said second slot position beingdetermined by the sequence number for said received fragments;

e) placing said head pointer in said second slot position; and

f) repeating steps a)-e) until a sequence of received head pointersstored in said second tier array indicates a complete sequence,

wherein

a complete sequence is a sequence of head pointers that does not have agap and includes a fragment that is an end fragment for said packet.

In a second aspect, the present invention provides a method ofreordering packet fragments, said packet fragments being derived from apacket which has been divided into at least two fragments, each fragmenthaving a sequence number denoting a position in a proper sequence forcorrect ordering of each fragment for a reassembly of said packet, eachfragment being represented by a head pointer, the method comprising:

a) receiving a received head pointer for a received fragment;

b) determining a sequence number for said received fragment;

c) determining a slot position for said received head pointer in apointer array, said slot position being determined by the sequencenumber for said received fragments;

d) placing said head pointer in said slot position;

e) repeating steps a)-d) until a sequence of received head pointersstored in said array indicates a complete sequence,

wherein a complete sequence is a sequence of head pointers that does nothave a gap and includes a fragment that is an end fragment for saidpacket.

In a third aspect, the present invention provides a method ofresequencing fragments forming portions of a data packet into a propersequence for a reassembly of said packet, each fragment having asequence number denoting a proper placement of said fragment in saidreassembly, the method comprising:

a) receiving a received head pointer for a received fragment, said headpointer representing said received fragment;

b) determining a reassembly session number for said received fragment,said reassembly session number denoting a home packet of which saidreceived fragment is a part;

c) determining if said home packet of which said received fragment is apart is currently being reassembled;

d) if said home packet is currently being reassembled, determining whichpointer array is being used in a reassembly of said home packet andplacing said head pointer in said pointer array;

e) if said home packet is not currently being reassembled, initiating areassembly of said home packet by allocating a pointer array for saidreassembly of said home packet;

f) determining a slot position for said head pointer in said pointerarray, said slot position being determined by the number for saidreceived fragment;

g) placing said head pointer in said slot position; and

h) repeating steps a)-g) until a sequence of received head pointersstored in said array indicates a complete sequence,

wherein a complete sequence is a sequence of head pointers that does nothave a gap and includes a fragment that is an end fragment for saidpacket.

In a fourth aspect, the present invention provides a method ofprocessing data frames in a multiple channel system, said data framescontaining a payload, said payload comprising portions of a larger datatransmission unit, the method comprising:

a) receiving a data frame;

b) determining with which channel said data frame is associated;

c) extracting a payload of said data frame;

d) storing said payload at a memory location specifically associatedwith said channel;

e) if previous payloads already stored at said memory location,appending said payload to said previous payloads; and

f) repeating steps a)-e) until a specific condition is met, saidspecific condition being chosen from the group comprising:

-   -   an amount of data stored in said memory location reaches a        predetermined value; and    -   a payload received and data stored in said memory location        relate to different data transmission units.

In a fifth aspect, the present invention provides a method of rebuildingfragments each of said fragments being a data transmission unit havingas payload a portion of data packet, said method comprising:

a) receiving a first chunk of data comprising a first segment of afragment;

b) storing said chunk of data in a first block of memory;

c) receiving a second chunk of data comprising a second segment of saidfragment;

d) placing said second chunk of data in a second block of memorycontiguous to said first block of memory;

e) linking said second contiguous block of memory with said firstcontiguous block of memory; and

f) repeating steps a)-e) until said fragment is complete,

wherein

said first chunk and said second chunk are received in a sequence whichpreserves a data integrity of said fragment.

In a sixth aspect, the present invention provides a fragment for use intransmitting data between two nodes in a network, said fragments having:

payload comprising a portion of a data packet;

a sequence number denoting a proper placement of said fragment in asequence of fragments when said data packet is being reassembled, and

a reassembly session number identifying which data packet said fragmentis a part of.

In a seventh aspect, the present invention provides a packet for use intransmitting data between two nodes in a network, and for use in amultiple link data transmission system, said packet having a connectionidentifier denoting to which grouped of links said packet belongs to.

In an eighth aspect, the present invention provides a system forrebuilding fragments each of said fragments being a data transmissionunit having as a payload a portion of data packet, the systemcomprising:

a memory bank for storing data chunks, each data chunk being a portionof a fragment;

a linked list pointer table, said pointer table having a record of whichspecific memory locations in said memory bank are used by specific datachunks

wherein

each data chunk relating to a specific fragment is stored in a specificmemory location, an address of such specific memory location beingplaced in said pointer table and linked to addresses of previouslystored data chunks.

In a ninth aspect, the present invention provides a system forresequencing fragments forming portions of a data packet, each of saidfragments having a sequence number denoting a proper placement of saidfragment in said packet and each of said fragments having a reassemblysession number denoting a home packet of which said fragment is a part,the system comprising:

at least one pointer array for each resequencing operation for storinghead pointers, each head pointer representing a received fragment, eachhead pointer being associated with a slot in one of said pointer arraysbased on the sequence number of a fragment represented by said headpointer; and

a lookup engine for determining if a fragment represented by a headpointer has a reassembly session number matching a resequencingoperation being performed,

wherein

in the event a fragment has a reassembly session number matching aresequencing operation being performed, its head pointer is placed in apointer array for said resequencing operation.

In a tenth aspect, the present invention provides a system forprocessing data frames in a multiple channel data transmission system,said data frames containing a payload, said payload comprising portionsof a larger data transmission unit, the system comprising:

a data frame engine for extracting a payload from said data frames;

a partial packet processor for storing payloads extracted from said dataframes;

an internal memory bank controlled by said processor such that saidpayloads are stored in said memory bank,

wherein

a specific memory location in said memory bank is allocated for eachchannel such that payloads from a specific channel are stored in saidmemory location.

In an eleventh aspect, the present invention provides a method ofselecting a link on which to transmit data in a multiple link system,the method comprising:

a) determining an amount of data queued for transmission on each link insaid multiple link system;

b) determining which link in said multiple link system has the most dataqueued for transmission;

c) selecting a link with a least amount of data queued for transmissionas the link on which to transmit data; and

d) if all links in said multiple link system have equal amounts of dataqueued for transmission, selecting each link in sequence as the link onwhich to transmit data.

In a twelfth aspect, the present invention provides a multiple stagesystem for processing data stream on a multiple link system, said systemcomprising:

a first stage for receiving data frames and extracting and storingpayloads from said data frames;

a second stage for rebuilding fragments from said payloads of said dataframes; and

a third stage for resequencing said fragments for eventualretransmission to a high speed data link,

wherein

each of said fragments forms a portion of a data packet,

each of said fragments having a sequence number denoting a properplacement of said fragment in said reassembly and each of said fragmentshaving a reassembly session number denoting a home packet of which saidfragment is a part.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the invention may be obtained by reading thedetailed description of the invention below, in conjunction with thefollowing drawings, in which:

FIG. 1 is a block diagram of a multiple link data transmission system;

FIG. 2 illustrates one format for fragment encapsulation;

FIG. 2A illustrates a format for an HDLC frame;

FIG. 3 is a block diagram of the internal components of the receive nodeof FIG. 1;

FIG. 4 illustrates a sample pointer table which may be used in a datastorage scheme according to the invention;

FIG. 5 illustrates a two tier data structure according to the invention;

FIG. 6 illustrates a second tier array and how it is populated by asequence of fragments arriving out of order;

FIG. 7 is a block diagram for a transmit system which transmits from ahigh speed line to a multilink bundle;

FIG. 8 is a flow chart which illustrates steps in a process executed inby the system illustrated in FIG. 3;

FIG. 9 is a flow chart detailing the steps executed in resequencing afragment using the two tier data structure illustrated in FIG. 5;

FIG. 10 details steps taken to retrieve data from the data stream; and

FIG. 11 details the steps involved in reassembling a fragment.

DETAILED DESCRIPTION

To clarify the following description, a brief description of the processexecuted on a packet to be sent on a multiple link bundle system will bedescribed. It should be noted that a multiple link bundle is acollection of data links between two nodes with the links beingcollectively used to increase bandwidth by utilizing them as if theywere a single link.

Referring to FIG. 1, a user 10 is connected to a transmit node 20. Thetransmit node 20 is connected to receive node 30 by way of “low” speedlinks 40A, 40B, 40C, 40D, 40E. The receive node 30 is coupled to a highspeed link 50. It should be noted that the terms “high speed” and “lowspeed” are relative to each other and are only used for illustrationpurposes. Thus, each of the links 40A, 40B, 40C, 40D, 40E can be T1lines transmitting at 1.544 mbps while the link 50 can be a T3 lineoperating at around 45 mbps. It should be noted that while T1 lines areused in this example, other types and speeds of links may be used.

In operation, the bundling solution works thus: the user 10 transmits apacket for eventual transmission on the link 50. The packet is receivedby the transmit node 30 and is divided into fragments. These fragmentscan then be independently transmitted via lines 40A, 40B, 40C, 40D, 40Eto the receive node 30. The receive node 30 then reassembles thereceived fragments to recreate the original packet. This original packetis then transmitted through the link 50 to its eventual destination.

The fragmentation process that the packet undergoes and the resultingfragments are generally compliant with existing standards such as RFC1990 and FRF 12/16.

Each portion of the fragmented packet is encapsulated within a specificencapsulation format, the packet fragment data being bracketed byfragment identification data. Each fragment transmitted by the transmitnode 20 is given a unique sequence number such that fragments resultingfrom a packet will have consecutive sequence numbers part of thefragment identification data, a fragment may also have an endingfragment bit set. Such a fragment denotes the ending fragment or thelast fragment in a sequence for a particular packet. Thus, if a packetwas fragmented into fragments with sequence numbers a, a+1, a+2, a+3,then fragment a+3 would have its ending fragment bit set as it is thefinal fragment for a particular packet. One possible format for thefragment encapsulation is illustrated in FIG. 2. The encapsulationformat could include an address file 60 to denote the destinationaddress of the fragment, a control field 70, an MP (multilink protocol)header field 80 which can include a beginning fragment bit 90 and anending fragment bit 100. Also included in the format would be thesequence number 110, the payload 120 or the actual portion of thepacket, and an FCS (Frame Check Sequence) field 130 for error correctionpurposes. In this specification the term “fragment” will be taken tomean an encapsulated portion of a packet that has been divided orfragmented into separate portions. The term “portion” will be taken tomean a section or part of a packet. Thus, a fragment will carry withinit a portion of a packet. To further illustrate the concept, thefragment is analogous to an egg while the portion will be analogous tothe egg yolk within the egg.

While not defined in the generally accepted standards for multiple linkapplications, a header field in each fragment would be quite useful inresolving the problems associated with multiple link bundling. Such aheader field, perhaps taking the place of the control field 70 in FIG.2, would include a reassembly session number. This reassembly sessionnumber identifies a reassembly session for a particular packet. Thisnumber can be assigned to a fragment when a packet is fragmented ordivided. Thus packet A can be assigned reassembly session number y andbe fragmented into a portions carried by fragments having sequencenumbers f1, f2, f3, and f4. Packet B, on the other hand, can be assignedthe unique reassembly session number z and be fragmented into portionscarried by fragments with sequence numbers f5, f6, f7, f8, f9, and f10.Reassembly of these two packets can therefore be simplified as it shouldnow be clear that fragments belonging to packet A will have a reassemblysession number y and that the proper sequencing of the portions carriedby these fragments is f1, f2, f3, and f4. Similar logic can be appliedto packet B when retrieving and resequencing its fragments.

As an important side note, it must be mentioned that not all packetsneed to be fragmented or divided. Some packets are of sufficiently smallsize that they can properly fit as payload for an encapsulated fragment.Clearly, such packets, when received by the receive node 30, need not beresequenced nor reassembled.

For the actual data transfer between the transmit node 20 and thereceive node 30, data is transferred using bit streams. A bitstream cancarry multiple channels and, to separate the data between channels, theHDLC (High Level Data Link Control) protocol is commonly used. Thisprotocol provides a separate frame for each channel's data. The framedelineates the beginning and the end in the bitstream for a channel'stransmission. A standard HDLC frame may have a format as illustrated inFIG. 2A. An opening flag field 131 denotes the beginning of the framewhile an address field 132 denotes the destination of the frame. Anextra address field 133 may be present, depending on the implementationof the protocol. A data section 134 will contain the data or payload ofthe frame while a CRC (Cyclic Redundancy Check) field 135 is present forerror checking purposes. Finally, a closing flag field 136 delineatesthe end of the frame.

Depending on the implementation of the protocol, the data field in theHDLC frame may contain a fragment or a packet. As an example of how thetransmit side may work, packet A is to be transmitted. Packet A willthus be divided into, for example, 3 portions corresponding to fragmentswith sequence numbers f1, f2, f3. Each fragment may be placed into asingle HDLC frame and transferred down a given link.

The sequence of these frames is preserved as each frame sequence,denoting a fragment, is dedicated to a single channel. Each channel isdedicated to a single physical link. Using this scheme, each HDLC framesequence is transmitted in sequence and, hence, the ordering of thefragment portions is preserved.

Referring to FIG. 3, a block diagram of the internal components of thereceive node 30 is illustrated. For ease of reference and explanation,this system 140 is divided into 3 sections: a receive/chunk retrievalsection 150, a fragment rebuilding section 160, and a fragmentresequencing/packet reassembly section 170.

The receive/chunk retrieval section 150 receives the bitstream of thedifferent links 40A, 40B, 40C, 40D, 40E at a channelizer 180. Thechannelizer 180 separates the channel in the bitstream and passes thechannelized bitstream to the HDLC engine 190. The channelizer 180associates the data in the bitstream with a particular channel. For eachlink, this is done by a serial to parallel conversion as serial datafrom the bitstream is parallelized by associating it with a particularHDLC channel. The HDLC engine examines each channelized bitstream anddetermines the beginning and end of each HDLC frame. Bit de-stuffing,error checking, and minimum/maximum size checking is performed by theHDLC engine 190. The payload or data carried by each HDLC frame, alongwith address and status information for that data, is passed by the HDLCengine 190 to the partial packet processor 200. The partial packetprocessor 200 controls a relatively small amount of RAM (random accessmemory) 210 that is preferably of the SRAM (static random access memory)type. This provides speed to the system. This RAM is used as a temporarystorage or a gathering area for data received for each HDLC channel.

Within this RAM 210, sections are associated for each HDLC channel andit is within these sections that data from the associated channel istemporarily stored up to a certain amount. Once this threshold amount isreached, the data is transferred to another, larger RAM bank for use bythe next section, the fragment rebuilding section 160.

In one implementation, the RAM 210 used by the partial packet processor200 is comprised of 64 kb of SRAM. Data associated with an HDLC channelis written to a location in the SRAM that is associated with thatchannel. Further data from that channel is aggregated with previouslyreceived data until a threshold, in this case 32 bytes, is reached. Itshould be noted that within this threshold (a “chunk” of data) data fromdifferent fragments or packets are not commingled. This commingling isprevented by using a flag that is present at the tail end of an HDLCframe sequence. The flag, when encountered, denotes that the HDLC framehaving the flag set is the last in the sequence and, hence, that thenext HDLC frame belongs to another sequence.

The fragment rebuilding section 16 controls another memory bank 220. Thefragment rebuilding section 160 receives chunks of data from the partialpacket processor 200 and stores these in its RAM bank 220. Fragmentsthat have been divided for transport between the transmit node 20 andthe receive node 30 are rebuilt in this section. The fragment rebuildingsection has logic which examines the data in the received chunks andrecognizes the beginning and the end sections of a fragment. One methodof accomplishing this is by having a set fragment size. The beginningsection is easy to find as it contains the fragment headers. Once thissection has been received, a running total of the accumulated fragmentsections for a given fragment will provide the size of the fragment.Once the set fragment size is reached, the fragment is consideredcomplete. Alternatively, a flag may be set in the final fragment sectionto denote that a fragment has been completed.

Regarding the data storage in the RAM bank 220, a linked list of chunksreceived for a particular fragment provides a simple yet efficient datastructure. As an example, a pointer table can be used to keep track ofthe chunks received and to which fragment a specific chunk belongs.Since data is received in chunks of fixed size from the partial packetprocessor, the RAM bank 220 can be delineated into sections having thesame size as a chunk and a pointer table denoting the beginning memoryaddress for a memory section can be used. The memory accesses, whenrequired, need only determine the beginning address of a chunk in RAMbank 220 and read from that beginning address to beginningaddress+chunk_size. Any new chunk of data can be added to the existinglinked list by appending its pointer to the tail of a linked list. FIG.4 illustrates such a pointer table.

Referring to FIG. 4, a pointer table which may be used in the abovescheme is illustrated. As can be seen, the first column has pointeraddresses while the second column has the beginning memory addresses foreach chunk of data. The final column has the addresses of the nextpointer in the linked list. To illustrate how the scheme may be used, wecan assume that Pointer A is unused. An incoming chunk (assumed to be abeginning section of a fragment) is thus stored in the memory areabeginning with memory address 000 pointed to by Pointer A. Assuming asecond chunk of data for the same fragment is now received, this chunkmay be placed at the memory area pointed by Pointer B. Thus, in thethird column for Pointer A, the pointer address for Pointer B isentered. The result is a two element linked list with the memory area ofPointer A linked with the memory area of Pointer B. Once a final sectionfor a fragment is received and placed in the list, there can be no linkafter the last chunk. Thus, the third column for that pointer/chunkwould have a terminator (perhaps a null) to denote a fragment end. If apointer's memory area is unused, the third column for that pointer wouldindicate its unused or free state.

Again referring to FIG. 4, it can be seen that there are three linkedlists represented in the table. The first list comprises (in order) PtrA→Ptr B→Ptr C→Ptr D (end). The second list comprises (in order) PtrE→Ptr G→Ptr J→Ptr K (end). The third list comprises (in order): PtrH→Ptr I (end).

Referring to FIG. 3, it should be noted that the fragment rebuilder 230also works on a channelized basis. This means that the fragmentrebuilder 230 can only work on one fragment or job per channel. For theexample illustrated in FIG. 4, it should be clear that each of thecompleted fragments is from a different channel to currently coexistwithin the control of the fragment rebuilder section 160.

Once a fragment is completed by the fragment rebuilder 230, control ofthat fragment (i.e. the header for that fragment, its status andinformation regarding it, and the first pointer for the fragment) ispassed to the fragment resequencing/packet reassembly section 170. Thissection gathers the data and the pointers for the different fragmentsand reassembles the packets from the fragments. This section alsorecognizes and processes packets that have not been fragmented. Theresequencing/packet reassembly section also 170 can be comprised of aheader lookup module 240, a resequencing module 250, and a frame/packetrebuild module 260.

The header lookup module 240 deals with the header field of eachreceived fragment. This header field, briefly explained above,identifies the reassembly session, the sequence number, and an end bitfor each fragment. As noted above, each reassembly session numberuniquely identifies a packet that has been fragmented. When a fragmentarrives at the lookup module 240, the reassembly session number ischecked against packet reassembly sessions that are in operation. If thereassembly session number is new and resequencing is required, areassembly session is initiated and resources are allocated to thissession. If the reassembly session is not new, then the fragment isdirected to the session to which the fragment belongs in theresequencing module.

It should be noted that the transfer of a fragment from the fragmentrebuilder 230 to the lookup module 240 is simply a matter of sending thehead pointer or first pointer of a fragment's linked list to the lookupmodule. For the lookup module to read the header in the fragment(usually the first few bytes of a fragment), the lookup module merelyretrieves the first few bytes of the fragment using the head or firstpointer for that fragment. The actual data of the fragment stays withinthe RAM bank 220. Only the pointer to that data is sent to the lookupmodule.

At this point it must be noted that the operation and components of thesystem heretofore described depends on the protocol used by the highspeed link 50 illustrated in FIG. 1. If the link 50 is packet or framebased, then the frame/packet builder 260 is active and receives theoutput of the resequencing module 250. This frame/packet builder thensends its output (properly reconstructed packets or frames as needed) tothe ingress queue manager 270. This ingress queue manager 270 managesqueue for the link 50. If, on the other hand, the link 50 is not packetbased and is merely a conduit transmitting a data stream of fragments,then the output of the resequencing module 250 is directly fed to theingress queue manager 270. In this case, the output of the resequencingmodule 250 would be a correctly sequenced stream of fragments.

To further explain the process executed by the resequencing module 250,it must be recalled that each fragment transmitted from the transmitmode 20 is assigned a unique and increasing sequence number. Thissequence number and the reassembly session number assigned to eachpacket determine if resequencing is required. Since sequence numbers aresequential and increasing, the resequencing module 250 expects aspecific sequence number from the lookup module 240. If an incomingfragment has the expected sequence number, then no resequencing isrequired as the resequencing module 250 is receiving the fragments inthe same sequence that the transmit node has transmitted them. Such afragment would thus be directed to the next stage, either thepacket/frame builder 260 or the ingress queue manager 270. If, on theother hand, the incoming fragment had a sequence number which did notmatch the expected sequence number, then reassembly would be required.If this offending fragment had a reassembly number matching a reassemblysession in progress, then the fragment is directed to that reassemblysession. If the reassembly session number is new, then a new reassemblysession is initiated and resources are allocated.

Once the resequencing module 250 determines that resequencing isrequired, the fragment is enqueued for resequencing. The enqueue processused for resequencing uses a two-tiered data structure to allow thehandling of large numbers of fragments. This data structure isillustrated schematically in FIG. 5. Both the first tier 280 and thesecond tier 290 in the data structure are arrays of pointers with statusbits. The first tier 280 is an array of pointers with each pointerpointing to a second tier array of pointers. The second tier 290 arrayhas entries that are head pointers or the first pointers in the linkedlist of pointers for a fragment. The placement of a head pointer for afragment depends on the sequence number for that fragment. In oneimplementation the placement in the first tier for a fragment isdetermined by applying the formula:location_(—)1=channel_offset+(seq #−head_seq #)/128 where channel_offsetis a given offset value to segregate channels

-   -   Seq # is the sequence number for the fragment head_seq # is the        sequence number for the fragment that begins the sequence for a        specific packet reassembly session, alternatively, this can be        the expected sequence number.

Similarly, the placement of a fragment in the second tier can use theformula:Location 2=channel_offset+(seq #−head_seq. #>>7)

Each first tier entry has, as a minimum, a pointer validation bit 292and a pointer 294 to a second tier array. The pointer validation bit 292is set if the pointer 294 to the second tier array is a valid pointer.For each secondary tier entry, two extra bits are added to the memoryaddress of the head of the fragment linked list. These two bits are theOCCUPIED bit 296 and the END bit 298. The OCCUPIED bit 296 is set if thesecond tier entry is occupied by a valid memory address for a headpointer 299 of a fragment linked list. The END bit 298 is set if thefragment referenced by the second tier entry is an end fragment of apacket. Whether the fragment is an end bit can easily be found byexamining the header of that fragment as an end bit exists under the RFC1990 protocol (see above discussion on the fragment format).

As an example of the above scheme, we can assume that a fragment havingthe following header values has arrived at the resequencing module:

Sequence #: 015

Head Pointer: A

End bit value: 0

Re-Assembly Session #: 101

If we further assume that the expected sequence number is 010, thereassembly module will see that the fragment does not have the expectedsequence number. If we assume that the reassembly session number 101 iscurrently in progress, then the reassembly module will retrieve the tierpointer arrays for that reassembly session. If we further assume that apreviously received fragment with sequence number 013 has been received,then this fragment will have its head pointer in the retrieved secondtier pointer array. The recently received fragment will thus be placedin the second tier array two locations away from the fragment with thesequence number 013. For the recently received fragment, the occupiedbit is to be set while the end bit is not. In the address portion of theentry, the value of A is to be placed.

It should be noted that fragments may arrive out of order and the lastfragment of a packet, a fragment with its end bit set, may not be thelast fragment received for a packet. It is specifically for suchinstances that the resequencing module rearranges the fragments so thattheir proper order may be restored. Basically, fragments arriving out oforder are slotted into their proper locations and, when the propersequence is restored, the group of fragments may be dispatched to thenext module.

To illustrate the above, FIG. 6 illustrates a sequence of fragmentsarriving out of order and being reordered. For the figure, the expectedsequence number is x and the fragment arrival sequence (using thefragment sequence numbers) is thus: x+5(END fragment), x+2, x+4, x+3,x+1, x. As can be seen in FIG. 6, the pointer array is gradually beingpopulated by the arriving fragments. As noted above, the placement of afragment is determined by a fragment's sequence number. Using theformula given above for second tier placement, the fragment withsequence number x+5 is placed in the 5^(th) slot of the array. Thesecond arriving fragment (having a sequence number of x+2) is placed inthe second slot of the array.

Similarly, the other arriving fragments are in turn slotted into theirproper slot in the second tier array. The bottom illustration of FIG. 6shows that the section of the packet is complete as the fragments havebeen resequenced. Once all the fragments are in place, including thefragment with the END bit set, the segment is complete and ready fordispatch to the next stage. It should be recalled that the expectedsequence number is x. Once the segment is complete, and after theexpected sequence number arrives, then the segment can be dispatched.After the fragment with the sequence number x is received, the nextexpected number is x+1. Since this fragment is in a completed segment,then this segment can be dispatched.

As a criteria for completeness, a collection of resequenced fragmentsmust be in order and be valid. This is determined by merely checking thestatus bits of the second tier array for a particular resequencingoperation. All the OCCUPIED bits must be set for all the fragments inthe sequence and the final fragment in the sequence must have its ENDbit set. With these two criteria met, the fragments in the sequence canbe taken to be in order and complete for this reassembly session.

Reading out the data for a completed fragment sequence is similar to theprocess for passing on to the next stage a fragment with the expectedsequence number. It should be recalled that passing a fragment to thenext stage, either to the ingress queue manager or to the packet/framebuilder, is preparatory to transmitting the fragment, or a packetrebuilt from the fragment through the high speed link. Thus, it is theactual data comprising the fragment that needs to be passed on. Giventhe linked list data structure for storing the fragment data in theexternal RAM 220, this can easily be done. To pass the fragment and itsdata, the data must be read from the RAM bank 220. This merely involvestraversing the linked list and reading the data in the memory locationpointed to by the pointers in the linked list. Once the last link hasbeen traversed and the data read and passed to the next stage, thefragment has effectively been transmitted.

A similar operation to the above is executed when reading out or passinga reordered sequence of fragments from the resequencing module. Totransfer the reordered fragments, each of the fragments are sequentiallypassed to the next stage. The first pointer in the second tier pointerarray is read and the linked list pointed to is sequentially read inmuch the same way as a fragment is read for passing on. Once the firstfragment in the sequence is read, the status bits for the pointer entryin the pointer array are reset so the array can be reused. Then, thenext fragment is read out from memory in the same manner. Thus, eachfragment is sequentially transferred to the next stage by reading itsdata from the memory until all the fragments in the relevant second tierpointer array are dispatched and the pointer is ready for reuse.

To ensure a continuous stream of properly sequenced fragments, a “roam”operation is periodically executed by the resequencing module. A “roam”operation checks the various reassembly/resequencing sessions to see ifany are ready for dispatch. As explained above, this involves justchecking the status bits of the second tier pointer array. If, for aspecific second tier array, all the OCCUPIED bits are set and the endfragment is present, then that sequence of fragments is ready fordispatch. However, to preserve packet order, the fragment sequence mustbe one that the packet/frame builder module or one that the ingressqueue manager requires. As an example, if the packet/frame buildermodule has packets A, B, and C currently being reassembled or built,fragment sequences that are not needed by those packets cannot bedispatched. If, on the other hand, a completed fragment sequence willcomplete packet B, then that fragment sequence can be dispatched ordequeued for passing on to the packet/frame builder. Similarly, if theresequencing module output is to be sent directly to the ingress queuemanager, then it may be the sequence number that determines whichfragment sequence is to be dispatched or dequeued. As an example, iffragment sequences with sequence numbers x1-x7, x10-x19, and x21-x25 aresequenced properly and ready for dispatch but the ingress queue managerneeds fragment x0, then none can be dispatched. Once fragment x0 isreceived and dispatched, then fragment sequence x1-x7 can be dispatchedas it is the next in the numbering sequence. Sequence x10-x19 cannot bedispatched until fragments x8-x9 arrive or are determined to be lost.While the above scheme notes using a two-tier system, implementationusing only a single tier is possible. Using only the second tier willallow resequencing of the fragments but this may not allow for largenumbers of fragments. Essentially the number of fragments per sequencewill be limited to the number of entries in a second tier array. If moreentries are required, a first tier can be used, thereby multiplying thenumber of fragments that can be accommodated by the number of entries inthe first tier array.

The example above notes a second tier array for a specific fragmentsequence. After a fragment sequence is read out of a secondary tier,this tier is placed in a pool of available arrays for use by otherfragments or fragment sequences. Also, while the examples above onlyshow a single fragment sequence in a second tier array, multiplefragment sequences can have a single second tier array. This is evidentfrom the sample formula given above for determining a fragments' placingin a second tier. Since fragments are given different channels, thechannel offset value in the formula allows for multiple channels, andhence multiple fragment sequences, to coexist in a single second tierarray.

It should be noted that either the header lookup module 240 or theresequencing module 250 can be equipped with the capability ofdetermining whether a specific fragment is to be considered as lost.This loss detection capability can be implemented by executingwell-known loss detection methods. These can be as simple as a time-outperiod for fragments with expected sequence numbers. If the expectedfragment does not arrive within a set period of time, the fragment canbe declared lost. This capability prevents the system from indefinitelywaiting for a fragment that may be lost due to circumstances beyond thesystem's or a user's control.

The next stage in the system, assuming the output of the resequencingmodule is not sent directly to the ingress queue manager 270, is thepacket/frame builder module 260. This module receives either fragmentswith the expected number or completed fragment sequences. Thepacket/frame builder module 260 aggregates fragments and removes thefragment “wrapper” or frame to extract the packet portion. These packetportions are then reassembled to arrive at the completed packet.

To help keep track of which packets are currently being assembled, thepacket/frame builder 260 takes note of the reassembly session number ofincoming fragments or fragment sequences. It must be recalled thatreordered fragment sequences from the resequencing module are reorderedor resequenced on a reassembly number basis. Once areassembly/resequencing operation in the resequencing module is complete(i.e. all fragments from a certain point are present and accounted for),the resulting fragment sequence is ready for dispatch. By merelymatching the reassembly session number of the fragment sequence with thereassembly session number of a packet being assembled in thepacket/frame building module, the packet can be mated with the rest ofits fragments. Thus, when the resequencing module executes a “roam”operation, it is equipped with the reassembly session number of thepackets being assembled in the packet/frame module. If a fragmentsequence is complete and its reassembly session number matches that of apacket being reassembled, these two are mated and the packet iscompleted.

Once the packets are assembled in the packet/frame builder module, theyare ready for the egress queue manager 270. This queue manager 270enqueues packets and/or fragments for transmission on the high speedlink 50.

While the above describes a system that transmits data from a multilinkbundle to a single high speed line, the concepts applied in the abovesystem can equally be used for a transmit system which transmits from ahigh speed line to a multi-link bundle. A block diagram of such a systemis illustrated in FIG. 7.

Referring to FIG. 7, the transmit system is comprised of a fragmentbuilder module 280, an ingress queue manager module 290, external RAM300, a transmit partial packet processor 310 HDLC transmit engine 320,internal RAM 330, and a de-channelizer module 340 which connects tomultilink bundle or bundles.

To explain the function of the different components, the fragmentbuilder 280 receives a data stream from the high speed line 50. Thisdata stream is composed of packets and the fragment builder 280 willretrieve the packets from the data stream. Not only that, but thefragment builder 280 will also fragment each packet. Each fragment ofeach packet will further be decomposed by the egress queue manager 290into smaller data units analogous to the concept of the “chunk”explained above. Each fragment, once properly created with the properformatting, headers, and sequence numbers, is placed in the external RAM300 using the linked list concept explained above. It should also benoted that, at this point, each fragment is assigned a specific channelby the egress queue manager.

With the incoming fragment stored in the external RAM 300 and properlydivided into convenient data units (“chunks”), the transmit partialpacket processor 310 will then retrieve specific amounts of these dataunits from the external RAM 300 as required by the transmit HDLC engine.As noted above, the HDLC protocol encapsulates smaller data units fortransmission via the multilink bundles.

It should be noted that the internal RAM 330 (usually fast SRAM) is usedby the transmit partial packet processor 310 in a similar fashion as theRAM 210 is used by the partial packet processor 200 in FIG. 3.Essentially this internal RAM 330 a temporary storage space for thesmaller units of data retrieved from the external RAM 300. The transmitpartial packet processor 310 retrieves a set amount of data from theexternal RAM 300 (say, 32 bytes) and places this in a designatedlocation in the internal RAM 330. When the transmit HDLC engine 320needs to send data through, the engine 320 or the partial packetprocessor 310 retrieves some or all the data in a specified location inthe internal RAM 330 and encapsulates it in an HDLC frame fortransmission.

It should be noted that the storage of data in the internal RAM 330 is,again, similar to that used for the RAM 210 in FIG. 1. Specific areas ofthe internal RAM 330 are set aside for specific channels. Thus, if thetransmit HDLC engine 320 needs to send data for channel A then thememory area for that channel is known and segregated.

As an improvement to existing systems, the incoming packets from thehigh speed link 50 in FIG. 7 may be equipped with a tag or connectionidentifier. This CI (connection identifier) identifies the multilinkbundle and the exact multi-link session associated with the packet.Similar in concept to the reassembly session number assigned tofragments as explained above, the CI allows multiple fragmentationsessions to occur simultaneously.

Returning to the transmit HDLC engine 320, this module receives the bytestream from the transmit partial packet processor 310 and encapsulatesor frames the data units with an HDLC header, bit stuffs the frame ifrequired, and appends an FCS (for error correction) trailer to theframe. These HDLC frames are then sent to the de-channelizer fortransmission on the multiple links.

Concerning the egress queue manager 290, this module can balance theload among the different links in a bundle. The egress queue manager 290determines the amount of data that is to be transmitted on each link anddetermines which link has the least amount of data to be transmitted orthe least data fill. An incoming frame or fragment for transmittal isthe assigned to this link. The egress queue manager 290 than repeats theprocess for each incoming frame, assigning the incoming frame to a linkthat has the least number of bytes to transmit. If all the links have anequal number of bytes to transmit, the de-channelizer can cycle throughthe links in a round-robin function.

The processes outlined about can be summarized in a few flowcharts forclarity. Referring to FIG. 8, a flowchart illustrates the steps in aprocess executed by the system in FIG. 3. Requiring with step 350, thatof receiving the data stream from the transmit mode, the process isinitialized. Step 360 is that of extracting the data contained in thedata stream. This data, in the form of HDLC frames, contains the dividedfragments. The data is then aggregated to form chunks of data in step370 using internal RAM banks with memory areas set aside for eachchannel. In step 380, the data chunks are aggregated to reassemble thefragments. As noted above, the chunks of data are passed from theinternal RAM banks to the external RAM and are sorted therein using thelinked list data structure explained above.

Once the fragment has been reassembled from the data chunks, step 390checks if the fragment has to be resequenced relative to the otherfragments already received. If the fragment is in need of resequencing,step 400 is that of resequencing the fragment by determining itsplacement in the sequence using its reassembly session number and itssequence number. Step 410 then determines if the fragment sequence canbe sent to the next stage. If the fragment sequence is not complete,then connector A shows that the logic flow returns to step 390 for a newfragment. If, on the other hand, the fragment sequence is complete orthe response to the query of step 390 is in the negative, then step 420checks if the high speed link at the end of the system uses packets or asimple data stream. Clearly, the answer to this step can be preset atthe system level. If the packets are not used, then step 430 is that ofsending the completed fragment sequence to the ingress queue manager. Ifpackets are used, step 440 is that of reassembling the packets from thecompleted fragment sequences for eventual transmission to the ingressqueue manager.

While the flowchart of FIG. 8 presents an overview of the actions of thesystem in FIG. 3, FIG. 9 details the steps executed in resequencing afragment using the two tier data structure outlined above. Thisresequencing process begins at step 450 with the reception of afragment. Once the fragment has been received, step 460 is determiningif the fragment has the expected sequence number. This determines if thefragment is the next expected fragment. If so, then step 470 detailssending the fragment to the next stage. If the fragment does not havethe expected sequence number, then step 480 queries whether thereassembly session number for the fragment is one that is already insession or in use. If the reassembly session is in use, then step 490details sending the fragment to the reassembly session with the samesession number for resequencing. Step 500 then enqueues the fragment forthe reassembly session. Step 510 then places the fragment head pointer(the first pointer for the linked list of memory locations containingthe data for the fragment) in the proper slot in a second tier arraybased on the sequence number. Step 520 then sets the status bits forthat end tier array entry accordingly. Connnector B then returns thelogic flow to step 450 to receive another fragment.

Returning to decision 480, if the reassembly session number is notcurrently in use, then step 530 details allocating resources for a newreassembly/resequencing session. This requires allocating a first tierarray and, when necessary, at least one second tier array (step 540).The head pointer for the fragment is then placed in the proper slot inthe tier array (step 550) and the status bits for this entry are thenset accordingly (step 560). Again, connector B details returning thelogic flow back to step 450.

Referring to FIGS. 10 and 11, these two flowcharts detail the steps inthe processes which lead to the reassembly of fragments from thedatastream received via the multiple links. The flowchart in FIG. 10recounts the steps taken to retrieve data from the datastream. Step 570initiates the process with the reception of an HDLC frame from thedatastream. The channel associated with the frame is then determined instep 580. The data contained in the frame is extracted in step 590. Step600 checks if the data received is for a different packet/fragment thanthe data already saved for that channel in the RAM bank. It should berecalled that specific memory locations and amounts are preallocated foreach channel in the RAM bank. If the data received indicates a newpacket/fragment, then step 610 reads out the data stored in memory fortransmission to the next stage in the system. Step 620 then saves thedata received at the memory location allocated for the channel.Connector C then notes that the process begins anew at step 570 with anew HDLC frame.

If the data received is not for a new packet/fragment, then step 630checks if enough data has been stored to constitute an acceptable chunkof data. If not then step 640 appends the data received to the datastored in memory. If there is sufficient stored data, then connector Dshows that the logic then flows to steps 610 and 620.

Referring to FIG. 11, the flowchart details the steps involved inreassembling a fragment. As can be seen, step 650 initiates the processby receiving a chunk of data from the internal RAM. Decision 660 thendetermines if the received chunk belongs to an existing fragmentbelonging to a specific channel. If there is no existing fragment, thena new fragment reassembly must be initiated. This is detailed in step670 in which the received chunk is sent to the external RAM and the headpointer for the newly initiated fragment is assigned to the pointer forthe memory location where the received chunk has been saved. This startsa new linked list for this fragment. Step 680 then checks if thefragment is complete. It should be noted that it is possible to have afragment consisting of only the chunk and this is why the next stepchecks for fragment completeness.

Returning to decision 660, if the chunk does belong to an existingfragment, then the received chunk is sent to external RAM for storage. Apointer to this location is appended to the linked list that detailswhere in external RAM the separate parts of the fragment are stored.Afterwards, step 680 checks for fragment completeness. If the fragmentis complete, then the header data for the fragment and the head pointerfor that fragment's linked list is sent to the next stage.

A person understanding the above-described invention may now conceive ofalternative designs, using the principles described herein. All suchdesigns which fall within the scope of the claims appended hereto areconsidered to be part of the present invention.

1. A method of selecting a link on which to transmit data in a multiplelink system, the method comprising: (a) determining an amount of dataqueued for transmission on each link in said multiple link system; (b)selecting a link with a least amount of data queued for transmission asthe link on which to transmit data; and (c) if all links in saidmultiple link system have equal amounts of data queued for transmission,selecting each link in sequence as the link on which to transmit data.2. The method of claim 1 further comprising, prior to step b):determining which link in said multiple link system has the most dataqueued for transmission.